Image data conversion

ABSTRACT

Embodiments of the present disclosure relate to an apparatus for converting image data from a Bayer format image to a four-plane image format using two memory channels. An example apparatus includes an interface for receiving the Bayer image including repeating pixel groups, where each pixel group includes a first pixel type, a second pixel type, a third pixel type, and a fourth pixel type. The apparatus also includes a memory and a circuit to write the Bayer image to the memory as four-plane data. The four-plane data includes pixels of the first type and the third type in the Bayer image that are written via the first memory channel, and pixels of the second type and the fourth type in the Bayer image that are written via the second memory channel. Embodiments also relate to converting three sensor image data to a Bayer format image using the two memory channels.

BACKGROUND

Image data captured by an image sensor or received from other datasources is often processed in an image processing pipeline beforefurther processing or consumption. For example, raw image data may becorrected, filtered, or otherwise modified before being provided tosubsequent components such as a video encoder. To perform corrections orenhancements for captured image data, various components, unit stages ormodules may be employed.

Such an image processing pipeline may be structured so that correctionsor enhancements to the captured image data can be performed in anexpedient way without consuming other system resources. Although manyimage processing algorithms may be performed by executing softwareprograms on central processing unit (CPU), execution of such programs onthe CPU would consume significant bandwidth of the CPU and otherperipheral resources as well as increase power consumption. Hence, imageprocessing pipeline is often implemented as a hardware componentseparate from the CPU and dedicated to perform one or more imageprocessing algorithms.

Images processed by the image processing pipeline may include severalimage data formats such as Bayer data format and three sensor dataformat (e.g., captured by three separate color sensors). Image datacorresponding to different image formats may need to be processed usingdifferent processing algorithms thereby increasing the complexity ofimage processing pipeline.

SUMMARY

Embodiments of the present disclosure relate to an apparatus and amethod for converting image data from a Bayer format image to afour-plane image format using two memory channels such as direct memoryaccess (DMA) channels. Embodiments also relate to reading the convertedfour-plane image format data back to the Bayer format image using thetwo memory channels.

In one embodiment, the apparatus includes an interface for receiving theBayer image in a format that includes repeating pixel groups, where eachpixel group is spread across a plurality of corresponding pixel rows andincludes a first pixel type, a second pixel type, a third pixel type,and a fourth pixel type. The apparatus also includes a memory and acircuit to write the Bayer image to the memory as four-plane data via afirst memory channel and a second memory channel. The four-plane datawritten to the memory includes pixels of the first type and pixels ofthe third type in the Bayer image that are written via the first memorychannel. The four-plane data written to the memory also includes pixelsof the second type and pixels of the fourth type in the Bayer image thatare written via the second memory channel.

In one embodiment, the four pixel types of each pixel group of the Bayerimage include a Red pixel for the first pixel type, Green pixel for eachof the second pixel type and the third pixel type, and a Blue pixel forthe fourth pixel type.

In one embodiment, the four pixel types of each pixel group of the Bayerimage include a Red pixel for the second pixel type, Green pixel foreach of the first pixel type and the fourth pixel type, and a Blue pixelfor the third pixel type.

In one embodiment, the four pixel types of each pixel group of the Bayerimage include a Red pixel for the first pixel type, Green pixel for eachof the second pixel type, White pixel for the third pixel type, and aBlue pixel for the fourth pixel type.

In one embodiment, the four pixel types of each pixel group of the Bayerimage include a Red pixel for the first pixel type, Green pixel for eachof the second pixel type, infrared (IR) pixel for the third pixel type,and a Blue pixel for the fourth pixel type.

In one embodiment, odd numbered pixel rows of the plurality ofcorresponding pixel rows of the Bayer image include alternating Red andGreen pixels, and even numbered pixel rows of the plurality ofcorresponding pixel rows of the Bayer image include alternating Greenand Blue pixels.

In one embodiment, the circuit further writes odd position pixels ofeach pixel row of the Bayer image to the memory via the first memorychannel and writes even position pixels of each pixel row of the Bayerimage to the memory via the second memory channel.

In one embodiment, the memory further includes two separate addressspaces, where each address space includes multiple sub-blocks. Eachsub-block of the multiple sub-blocks may store a contiguous datacorresponding to one pixel type of the image written to the memory,where the one pixel type may be one of the first pixel type, the secondpixel type, the third pixel type, and the fourth pixel type.

In one embodiment, the circuit is further configured to read pixels ofthe first type and the third type via the first memory channel, readpixels of the second type and fourth type via the second memory channel,and convert the read pixels to an image in the format that comprisesrepeating pixel groups, where the format is a Bayer format.

Embodiments further relate to converting three sensor image data to aBayer format image using the two memory channels. In one embodiment, theapparatus includes an interface for receiving a first image includingpixels of a first type, a second image including pixels of a secondtype, a third image including pixels of a third type. The apparatus alsoincludes a circuit for generating a Bayer format image from the firstimage, the second image, and the third image.

In one embodiment, the apparatus further includes a Red image sensor forcapturing the first image including Red pixels, a Green image sensor forcapturing the second image including Green pixels, and a Blue imagesensor for capturing the third image including Blue pixels.

In one embodiment, the circuit further generates the Bayer format imageby combining data corresponding to pixel rows of the first image and thesecond image to generate odd numbered pixel rows of the Bayer formatimage, and combining data corresponding to pixel rows of the secondimage and the third image to generate even numbered pixel rows of theBayer format image.

Embodiments also relate to a non-transitory computer-readable mediumstoring a digital representation of an example apparatus for convertingimage data from a Bayer format image to a four-plane image format,reading the converted four-plane image format data back to the Bayerformat image, and converting three sensor image data to a Bayer formatimage.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a high-level diagram of an electronic device, according to oneembodiment.

FIG. 2 is a block diagram illustrating components in the electronicdevice, according to one embodiment.

FIG. 3 is a block diagram illustrating image processing pipelineimplemented using an image signal processor, according to oneembodiment.

FIG. 4 is a block diagram illustrating conversion of image data fromBayer data format to four-plane data format, according to oneembodiment.

FIG. 5 is a block diagram illustrating additional details for convertingimage data from Bayer data format to four-plane data format, accordingto one embodiment.

FIG. 6 is a flowchart illustrating a method of converting image datafrom Bayer data format to four-plane data format, according to oneembodiment.

FIGS. 7A-7C illustrate modified Bayer image data formats, according toone embodiment.

FIG. 8 is a block diagram illustrating conversion of image data fromthree sensor data format to Bayer data format, according to oneembodiment.

FIG. 9 is a block diagram illustrating additional details for convertingimage data from three sensor data format to Bayer data format, accordingto one embodiment.

FIG. 10 is a flowchart illustrating a method of converting image datafrom three sensor data format to Bayer data format, according to oneembodiment.

The figures depict, and the detail description describes, variousnon-limiting embodiments for purposes of illustration only.

DETAILED DESCRIPTION

Reference will now be made in detail to embodiments, examples of whichare illustrated in the accompanying drawings. In the following detaileddescription, numerous specific details are set forth in order to providea thorough understanding of the various described embodiments. However,the described embodiments may be practiced without these specificdetails. In other instances, well-known methods, procedures, components,circuits, and networks have not been described in detail so as not tounnecessarily obscure aspects of the embodiments.

An embodiment of the present disclosure relates to image data conversionof images captured by an image sensor to a four-plane data format toincrease an image signal processor's compatibility with various imagedata formats. For example, image data of images captured in a Bayer dataformat is converted into a four-plane data format, where each plane ofdata corresponds to a different pixel type data of the Bayer data. Thethree sensor data format of images captured using three separate colorsensors is converted to Bayer data, which can further be converted tothe four-plane data for further image processing.

The image data conversion from the Bayer data to the four-plane dataincludes receiving image data in the Bayer data, where the receivedimage data includes repeating pixel groups. Each pixel group may be a2×2 block of pixels that is spread across a plurality of correspondingpixel rows. Each pixel group can includes a first pixel type such as Redpixels, a second pixel type such as Green pixels of odd pixel rows, athird pixel type such as Green pixels of even pixel rows, and a fourthpixel type such as Blue pixels. The conversion also includes writing theimage data to a memory via a first memory channel and a second memorychannel, where the pixels of the first type and the third type of theimage are written via the first memory channel, and the pixels of thesecond type and the fourth type in the image are written via the secondmemory channel.

The image data conversion from the three sensor data to the Bayer dataincludes receiving a first image including Red sensor data, a secondimage including Green sensor data, and a third image including Bluesensor data. The conversion also includes writing image datacorresponding to the first image, the second image, and the third imageto a memory via a first memory channel and a second memory channel,where the Red sensor data and the Green sensor data is written to thememory via the first memory channel, and where the Green sensor data andthe Blue sensor data is written to the memory via the second memorychannel.

Image data conversion from either Bayer data format to four-plane dataformat or three sensor data format to Bayer data format offers severaladvantages. Such conversion enables the same image signal processor(ISP) to be compatible with images captured in a variety of image dataformats such as three sensor data format, Bayer data format, or any ofthe modified Bayer data formats as illustrated in FIGS. 7A-7C. Byconverting any image captured in any format into the four-plane dataformat, an ISP designed to process the four-plane data format can alsoprocess image data captured in any of the above-mentioned data formats.Also, multiple sensors can be combined and processed as if they camefrom a conventional Bayer camera. For example, image data captured usedthree charge-coupled device (CCD) camera could be converted to Bayerformat data. Moreover, converting image data formats using two memorychannels (e.g., two DMA channels) over a single channel or even fourchannels have advantages. For example, using two channels providesincreased speed and efficiency compared to a single channel. Whencompared to four channels, the two-channel approach reduces resourceoverhead (e.g., memory buffers 414).

The term “Bayer format” or “Bayer data format” described herein refersto an image data format where the image data includes repeating pixelgroups as shown in Bayer data 510 of FIG. 5. Here, each pixel groupincludes four pixels arranged in a 2×2 block of pixels. For example, Redpixel 511, Green pixel 512, Green pixel 521, and Blue pixel 522 form apixel group. The Bayer data is generated using an image sensor thatincludes a Bayer color filter array (CFA) to captures light usingdifferent color filters.

The term “four-plane data format” refers to a data format where imagedata associated with different pixel types is stored as different planesof data. Here, a plane of data refers to a 2 dimensional matrix of datacorresponding to a particular type of pixel, such as a color of thepixel. For example, image data including four types of pixels is storedas four separate planes of data where each plane of data corresponds toeach pixel type. For example, the four-plane data 550 of FIG. 5converted from the Bayer data 510 includes four planes of data. Asdiscussed above, each pixel group of the Bayer data 510 includes a Redpixel, Green pixel Gr, Green pixel Gb, and Blue pixel. Each plane ofdata of the four-plane data 550 includes image data corresponding to oneof the four pixel types of the pixel groups of the Bayer data 510. Forexample, a first plane of data includes all Red pixels of the Bayer data510, a second plane of data includes all Green pixels (Gr) of odd pixelrows of the Bayer data 510, a third plane of data includes all Greenpixels (Gb) of the even pixel rows of the Bayer data 510, and a fourthplane of data includes all Blue pixels of the Bayer data 510. For theexample four-plane data 550, each individual plane of data is storedacross several memory sub-blocks, where each memory sub-blockcorresponds to a pixel row of Bayer data 510. For example, the plane ofdata for the Red pixels includes memory sub-block 581 corresponding tothe first pixel row of Bayer data 510 and memory sub-block 583corresponding to the third pixel row of Bayer data 510.

The term “three sensor data” described herein refers to image datacorresponding to an image captured using three separate color sensorssuch as a Red color sensor, Green color sensor, and Blue color sensor.An example three sensor data is shown in FIG. 9 as image datacorresponding to three images, Red sensor data 910, Green sensor data920, and Blue sensor data 930, captured by the three sensors.

Exemplary Electronic Device

Embodiments of electronic devices, user interfaces for such devices, andassociated processes for using such devices are described. In someembodiments, the device is a portable communications device, such as amobile telephone, that also contains other functions, such as personaldigital assistant (PDA) and/or music player functions. Exemplaryembodiments of portable multifunction devices include, withoutlimitation, the iPhone®, iPod Touch®, Apple Watch®, and iPad® devicesfrom Apple Inc. of Cupertino, Calif. Other portable electronic devices,such as wearables, laptops or tablet computers, are optionally used. Insome embodiments, the device is not a portable communications device,but is a desktop computer or other computing device that is not designedfor portable use. In some embodiments, the disclosed electronic devicemay include a touch sensitive surface (e.g., a touch screen displayand/or a touch pad). An example electronic device described below inconjunction with FIG. 1 (e.g., device 100) may include a touch-sensitivesurface for receiving user input. The electronic device may also includeone or more other physical user-interface devices, such as a physicalkeyboard, a mouse and/or a joystick.

FIG. 1 is a high-level diagram of an electronic device 100, according toone embodiment. Device 100 may include one or more physical buttons,such as a “home” or menu button 104. Menu button 104 is, for example,used to navigate to any application in a set of applications that areexecuted on device 100. In some embodiments, menu button 104 includes afingerprint sensor that identifies a fingerprint on menu button 104. Thefingerprint sensor may be used to determine whether a finger on menubutton 104 has a fingerprint that matches a fingerprint stored forunlocking device 100. Alternatively, in some embodiments, menu button104 is implemented as a soft key in a graphical user interface (GUI)displayed on a touch screen.

In some embodiments, device 100 includes touch screen 150, menu button104, push button 106 for powering the device on/off and locking thedevice, volume adjustment buttons 108, Subscriber Identity Module (SIM)card slot 110, head set jack 112, and docking/charging external port124. Push button 106 may be used to turn the power on/off on the deviceby depressing the button and holding the button in the depressed statefor a predefined time interval; to lock the device by depressing thebutton and releasing the button before the predefined time interval haselapsed; and/or to unlock the device or initiate an unlock process. Inan alternative embodiment, device 100 also accepts verbal input foractivation or deactivation of some functions through microphone 113. Thedevice 100 includes various components including, but not limited to, amemory (which may include one or more computer readable storagemediums), a memory controller, one or more central processing units(CPUs), a peripherals interface, an RF circuitry, an audio circuitry,speaker 111, microphone 113, input/output (I/O) subsystem, and otherinput or control devices. Device 100 may include one or more imagesensors 164, one or more proximity sensors 166, and one or moreaccelerometers 168. The device 100 may include components not shown inFIG. 1.

Device 100 is only one example of an electronic device, and device 100may have more or fewer components than listed above, some of which maybe combined into a components or have a different configuration orarrangement. The various components of device 100 listed above areembodied in hardware, software, firmware or a combination thereof,including one or more signal processing and/or application specificintegrated circuits (ASICs).

FIG. 2 is a block diagram illustrating components in device 100,according to one embodiment. Device 100 may perform various operationsincluding image processing. For this and other purposes, the device 100may include, among other components, image sensor 202, system-on-a chip(SOC) component 204, system memory 230, persistent storage (e.g., flashmemory) 228, orientation sensor 234, and display 216. The components asillustrated in FIG. 2 are merely illustrative. For example, device 100may include other components (such as speaker or microphone) that arenot illustrated in FIG. 2. Further, some components (such as orientationsensor 234) may be omitted from device 100.

Image sensor 202 is a component for capturing image data and may beembodied, for example, as a complementary metal-oxide-semiconductor(CMOS) active-pixel sensor) a camera, video camera, or other devices.Image sensor 202 generates raw image data that is sent to SOC component204 for further processing. In some embodiments, the image dataprocessed by SOC component 204 is displayed on display 216, stored insystem memory 230, persistent storage 228 or sent to a remote computingdevice via network connection. The raw image data generated by imagesensor 202 may be in a Bayer color filter array (CFA) pattern(hereinafter also referred to as “Bayer pattern”).

Motion sensor 234 is a component or a set of components for sensingmotion of device 100. Motion sensor 234 may generate sensor signalsindicative of orientation and/or acceleration of device 100. The sensorsignals are sent to SOC component 204 for various operations such asturning on device 100 or rotating images displayed on display 216.

Display 216 is a component for displaying images as generated by SOCcomponent 204. Display 216 may include, for example, liquid crystaldisplay (LCD) device or an organic light emitting diode (OLED) device.Based on data received from SOC component 204, display 116 may displayvarious images, such as menus, selected operating parameters, imagescaptured by image sensor 202 and processed by SOC component 204, and/orother information received from a user interface of device 100 (notshown).

System memory 230 is a component for storing instructions for executionby SOC component 204 and for storing data processed by SOC component204. System memory 230 may be embodied as any type of memory including,for example, dynamic random access memory (DRAM), synchronous DRAM(SDRAM), double data rate (DDR, DDR2, DDR3, etc.) RAMBUS DRAM (RDRAM),static RAM (SRAM) or a combination thereof. In some embodiments, systemmemory 230 may store pixel data or other image data or statistics invarious formats.

Persistent storage 228 is a component for storing data in a non-volatilemanner. Persistent storage 228 retains data even when power is notavailable. Persistent storage 228 may be embodied as read-only memory(ROM), flash memory or other non-volatile random access memory devices.

SOC component 204 is embodied as one or more integrated circuit (IC)chip and performs various data processing processes. SOC component 204may include, among other subcomponents, image signal processor (ISP)206, a central processor unit (CPU) 208, a network interface 210, sensorinterface 212, display controller 214, graphics processor (GPU) 220,memory controller 222, video encoder 224, storage controller 226, andvarious other input/output (I/O) interfaces 218, and bus 232 connectingthese subcomponents. SOC component 204 may include more or fewersubcomponents than those shown in FIG. 2.

ISP 206 is hardware that performs various stages of an image processingpipeline. In some embodiments, ISP 206 may receive raw image data fromimage sensor 202, and process the raw image data into a form that isusable by other subcomponents of SOC component 204 or components ofdevice 100. ISP 206 may perform various image-manipulation operationssuch as image translation operations, horizontal and vertical scaling,color space conversion and/or image stabilization transformations, asdescribed below in detail with reference to FIG. 3.

CPU 208 may be embodied using any suitable instruction set architecture,and may be configured to execute instructions defined in thatinstruction set architecture. CPU 208 may be general-purpose or embeddedprocessors using any of a variety of instruction set architectures(ISAs), such as the x86, PowerPC, SPARC, RISC, ARM or MIPS ISAs, or anyother suitable ISA. Although a single CPU is illustrated in FIG. 2, SOCcomponent 204 may include multiple CPUs. In multiprocessor systems, eachof the CPUs may commonly, but not necessarily, implement the same ISA.

Graphics processing unit (GPU) 220 is graphics processing circuitry forperforming graphical data. For example, GPU 220 may render objects to bedisplayed into a frame buffer (e.g., one that includes pixel data for anentire frame). GPU 220 may include one or more graphics processors thatmay execute graphics software to perform a part or all of the graphicsoperation, or hardware acceleration of certain graphics operations.

I/O interfaces 218 are hardware, software, firmware or combinationsthereof for interfacing with various input/output components in device100. I/O components may include devices such as keypads, buttons, audiodevices, and sensors such as a global positioning system. I/O interfaces218 process data for sending data to such I/O components or process datareceived from such I/O components.

Network interface 210 is a subcomponent that enables data to beexchanged between devices 100 and other devices via one or more networks(e.g., carrier or agent devices). For example, video or other image datamay be received from other devices via network interface 210 and bestored in system memory 230 for subsequent processing (e.g., via aback-end interface to image signal processor 206, such as discussedbelow in FIG. 3) and display. The networks may include, but are notlimited to, Local Area Networks (LANs) (e.g., an Ethernet or corporatenetwork) and Wide Area Networks (WANs). The image data received vianetwork interface 210 may undergo image processing processes by ISP 206.

Sensor interface 212 is circuitry for interfacing with motion sensor234. Sensor interface 212 receives sensor information from motion sensor234 and processes the sensor information to determine the orientation ormovement of the device 100.

Display controller 214 is circuitry for sending image data to bedisplayed on display 216. Display controller 214 receives the image datafrom ISP 206, CPU 208, graphic processor or system memory 230 andprocesses the image data into a format suitable for display on display216.

Memory controller 222 is circuitry for communicating with system memory230. Memory controller 222 may read data from system memory 230 forprocessing by ISP 206, CPU 208, GPU 220 or other subcomponents of SOCcomponent 204. Memory controller 222 may also write data to systemmemory 230 received from various subcomponents of SOC component 204.

Video encoder 224 is hardware, software, firmware or a combinationthereof for encoding video data into a format suitable for storing inpersistent storage 128 or for passing the data to network interface w10for transmission over a network to another device.

In some embodiments, one or more subcomponents of SOC component 204 orsome functionality of these subcomponents may be performed by softwarecomponents executed on ISP 206, CPU 208 or GPU 220. Such softwarecomponents may be stored in system memory 230, persistent storage 228 oranother device communicating with device 100 via network interface 210.

Image data or video data may flow through various data paths within SOCcomponent 204. In one example, raw image data may be generated from theimage sensor 202 and processed by ISP 206, and then sent to systemmemory 230 via bus 232 and memory controller 222. After the image datais stored in system memory 230, it may be accessed by video encoder 224for encoding or by display 116 for displaying via bus 232.

In another example, image data is received from sources other than theimage sensor 202. For example, video data may be streamed, downloaded,or otherwise communicated to the SOC component 204 via wired or wirelessnetwork. The image data may be received via network interface 210 andwritten to system memory 230 via memory controller 222. The image datamay then be obtained by ISP 206 from system memory 230 and processedthrough one or more image processing pipeline stages, as described belowin detail with reference to FIG. 3. The image data may then be returnedto system memory 230 or be sent to video encoder 224, display controller214 (for display on display 216), or storage controller 226 for storageat persistent storage 228.

Example Image Signal Processing Pipeline

FIG. 3 is a block diagram illustrating image processing pipelineimplemented using ISP 206, according to one embodiment. In theembodiment of FIG. 3, ISP 206 is coupled to image sensor 102 to receiveraw image data. ISP 206 implements an image processing pipeline whichmay include a set of stages that process image information fromcreation, capture or receipt to output. ISP 206 may include, among othercomponents, sensor interface 302, memory interface 326, central control320, front-end pipeline stages 330, back-end pipeline stages 340, imagestatistics module 304, vision module 322, back-end interface 342, andoutput interface 316. ISP 206 may include other components notillustrated in FIG. 3 or may omit one or more components illustrated inFIG. 3.

In one or more embodiments, different components of ISP 206 processimage data at different rates. In the embodiment of FIG. 3, front-endpipeline stages 330 (e.g., raw processing stage 306 and resampleprocessing stage 308) may process image data at an initial rate. Thus,the various different techniques, adjustments, modifications, or otherprocessing operations performed by these front-end pipeline stages 330at the initial rate. For example, if the front-end pipeline stages 330process 2 pixels per clock cycle, then raw processing stage 308operations (e.g., black level compensation, highlight recovery anddefective pixel correction) may process 2 pixels of image data at atime. In contrast, one or more back-end pipeline stages 340 may processimage data at a different rate less than the initial data rate. Forexample, in the embodiment of FIG. 3, back-end pipeline stages 340(e.g., noise processing stage 310, color processing stage 312, andoutput rescale 314) may be processed at a reduced rate (e.g., 1 pixelper clock cycle).

Sensor interface 302 receives raw image data from image sensor 202 andprocesses the raw image data into an image data processable by otherstages in the pipeline. Sensor interface 302 may perform variouspreprocessing operations, such as image cropping, binning or scaling toreduce image data size. In some embodiments, pixels are sent from theimage sensor 202 to sensor interface 302 in raster order (i.e.,horizontally, line by line). The subsequent processes in the pipelinemay also be performed in raster order and the result may also be outputin raster order. Although only a single image sensor and a single sensorinterface 302 are illustrated in FIG. 3, when more than one image sensoris provided in device 100, a corresponding number of sensor interfacesmay be provided in ISP 206 to process raw image data from each imagesensor.

Memory interface module 326 receives image data from the sensorinterface module 302 and provides the image data to the front-endpipeline stages 330. In one embodiment, the memory interface module 326provides the image data to the image statistics module 304 and thevision module 322. Memory interface module includes memory logic (e.g.,DMA logic) and planarizers such as planarizers 410 described below withreference to FIG. 4.

Front-end pipeline stages 330 process image data in raw or full-colordomains. Front-end pipeline stages 330 may include, but are not limitedto, raw processing stage 306 and resample processing stage 308. A rawimage data may be in Bayer raw format, for example. In Bayer raw imageformat, pixel data with values specific to a particular color (insteadof all colors) is provided in each pixel. In an image capturing sensor,image data is typically provided in a Bayer pattern. Raw processingstage 306 may process image data in a Bayer raw format or RGB format.

The operations performed by raw processing stage 306 include, but arenot limited, sensor linearization, black level compensation, fixedpattern noise reduction, defective pixel correction, raw noisefiltering, lens shading correction, white balance gain, and highlightrecovery. Sensor linearization refers to mapping non-linear image datato linear space for other processing. Black level compensation refers toproviding digital gain, offset and clip independently for each colorcomponent (e.g., Gr, R, B, Gb) of the image data. Fixed pattern noisereduction refers to removing offset fixed pattern noise and gain fixedpattern noise by subtracting a dark frame from an input image andmultiplying different gains to pixels. Defective pixel correction refersto detecting defective pixels, and then replacing defective pixelvalues. Raw noise filtering refers to reducing noise of image data byaveraging neighbor pixels that are similar in brightness. Highlightrecovery refers to estimating pixel values for those pixels that areclipped (or nearly clipped) from other channels. Lens shading correctionrefers to applying a gain per pixel to compensate for a dropoff inintensity roughly proportional to a distance from a lens optical center.White balance gain refers to providing digital gains for white balance,offset and clip independently for all color components (e.g., Gr, R, B,Gb in Bayer format). Components of ISP 206 may convert raw image datainto image data in full-color domain, and thus, raw processing stage 308may process image data in the full-color domain in addition to orinstead of raw image data.

Resample processing stage 308 performs various operations to convert,resample, or scale image data received from raw processing stage 306.Operations performed by resample processing stage 308 may include, butnot limited to, demosaic operation, per-pixel color correctionoperation, Gamma mapping operation, color space conversion anddownscaling or sub-band splitting. Demosaic operation refers toconverting or interpolating missing color samples from raw image data(for example, in a Bayer pattern) to output image data into a full-colordomain. Demosaic operation may include low pass directional filtering onthe interpolated samples to obtain full-color pixels. Per-pixel colorcorrection operation refers to a process of performing color correctionon a per-pixel basis using information about relative noise standarddeviations of each color channel to correct color without amplifyingnoise in the image data. Gamma mapping refers to converting image datafrom input image data values to output data values to perform specialimage effects, including black and white conversion, sepia toneconversion, negative conversion, or solarize conversion. For the purposeof Gamma mapping, lookup tables (or other structures that index pixelvalues to another value) for different color components or channels ofeach pixel (e.g., a separate lookup table for Y, Cb, and Cr colorcomponents) may be used. Color space conversion refers to convertingcolor space of an input image data into a different format. In oneembodiment, resample processing stage 308 converts RBD format into YCbCrformat for further processing.

Central control module 320 may control and coordinate overall operationof other components in ISP 206. Central control module 320 performsoperations including, but not limited to, monitoring various operatingparameters (e.g., logging clock cycles, memory latency, quality ofservice, and state information), updating or managing control parametersfor other components of ISP 206, and interfacing with sensor interface302 to control the starting and stopping of other components of ISP 206.For example, central control module 320 may update programmableparameters for other components in ISP 206 while the other componentsare in an idle state. After updating the programmable parameters,central control module 320 may place these components of ISP 206 into arun state to perform one or more operations or tasks. Central controlmodule 320 may also instruct other components of ISP 206 to store imagedata (e.g., by writing to system memory 230 in FIG. 2) before, during,or after resample processing stage 308. In this way full-resolutionimage data in raw or full-color domain format may be stored in additionto or instead of processing the image data output from resampleprocessing stage 308 through backend pipeline stages 340.

Image statistics module 304 performs various operations to collectstatistic information associated with the image data. The operations forcollecting statistics information may include, but not limited to,sensor linearization, mask patterned defective pixels, sub-sample rawimage data, detect and replace non-patterned defective pixels, blacklevel compensation, lens shading correction, and inverse black levelcompensation. After performing one or more of such operations,statistics information such as 3A statistics (Auto white balance (AWB),auto exposure (AE), auto focus (AF)), histograms (e.g., 2D color orcomponent) and any other image data information may be collected ortracked. In some embodiments, certain pixels' values, or areas of pixelvalues may be excluded from collections of certain statistics data(e.g., AF statistics) when preceding operations identify clipped pixels.Although only a single statistics module 304 is illustrated in FIG. 3,multiple image statistics modules may be included in ISP 206. In suchembodiments, each statistic module may be programmed by central controlmodule 320 to collect different information for the same or differentimage data.

Vision module 322 performs various operations to facilitate computervision operations at CPU 208 such as facial detection in image data. Thevision module 322 may perform various operations includingpre-processing, global tone-mapping and Gamma correction, vision noisefiltering, resizing, keypoint detection, convolution and generation ofhistogram-of-orientation gradients (HOG). The pre-processing may includesubsampling or binning operation and computation of luminance if theinput image data is not in YCrCb format. Global mapping and Gammacorrection can be performed on the pre-processed data on luminanceimage. Vision noise filtering is performed to remove pixel defects andreduce noise present in the image data, and thereby, improve the qualityand performance of subsequent computer vision algorithms. Such visionnoise filtering may include detecting and fixing dots or defectivepixels, and performing bilateral filtering to reduce noise by averagingneighbor pixels of similar brightness. Various vision algorithms useimages of different sizes and scales. Resizing of an image is performed,for example, by binning or linear interpolation operation. Keypoints arelocations within an image that are surrounded by image patches wellsuited to matching in other images of the same scene or object. Suchkeypoints are useful in image alignment, computing cameral pose andobject tracking. Keypoint detection refers to the process of identifyingsuch keypoints in an image. Convolution is heavily used tools inimage/video processing and machine vision. Convolution may be performed,for example, to generate edge maps of images or smoothen images. HOGprovides descriptions of image patches for tasks in mage analysis andcomputer vision. HOG can be generated, for example, by (i) computinghorizontal and vertical gradients using a simple difference filter, (ii)computing gradient orientations and magnitudes from the horizontal andvertical gradients, and (iii) binning the gradient orientations.

Back-end interface 342 receives image data from other image sources thanimage sensor 102 and forwards it to other components of ISP 206 forprocessing. For example, image data may be received over a networkconnection and be stored in system memory 230. Back-end interface 342retrieves the image data stored in system memory 230 and provide it toback-end pipeline stages 340 for processing. One of many operations thatare performed by back-end interface 342 is converting the retrievedimage data to a format that can be utilized by back-end processingstages 340. For instance, back-end interface 342 may convert RGB, YCbCr4:2:0, or YCbCr 4:2:2 formatted image data into YCbCr 4:4:4 colorformat. Alternatively, the back-end interface 342 may convert a Bayerdata format to a four-plane data format, the four-plane data format tothe Bayer data format, or a three sensor data format to the Bayer dataformat.

Back-end pipeline stages 340 processes image data according to aparticular full-color format (e.g., YCbCr 4:4:4 or RGB). In someembodiments, components of the back-end pipeline stages 340 may convertimage data to a particul4ar full-color format before further processing.Back-end pipeline stages 340 may include, among other stages, noiseprocessing stage 310 and color processing stage 312. Back-end pipelinestages 340 may include other stages not illustrated in FIG. 3.

Noise processing stage 310 performs various operations to reduce noisein the image data. The operations performed by noise processing stage310 include, but are not limited to, color space conversion,gamma/de-gamma mapping, temporal filtering, noise filtering, lumasharpening, and chroma noise reduction. The color space conversion mayconvert an image data from one color space format to another color spaceformat (e.g., RGB format converted to YCbCr format). Gamma/de-gammaoperation converts image data from input image data values to outputdata values to perform special image effects. Temporal filtering filtersnoise using a previously filtered image frame to reduce noise. Forexample, pixel values of a prior image frame are combined with pixelvalues of a current image frame. Noise filtering may include, forexample, spatial noise filtering. Luma sharpening may sharpen lumavalues of pixel data while chroma suppression may attenuate chroma togray (i.e. no color). In some embodiment, the luma sharpening and chromasuppression may be performed simultaneously with spatial nose filtering.The aggressiveness of noise filtering may be determined differently fordifferent regions of an image. Spatial noise filtering may be includedas part of a temporal loop implementing temporal filtering. For example,a previous image frame may be processed by a temporal filter and aspatial noise filter before being stored as a reference frame for a nextimage frame to be processed. In other embodiments, spatial noisefiltering may not be included as part of the temporal loop for temporalfiltering (e.g., the spatial noise filter may be applied to an imageframe after it is stored as a reference image frame (and thus is not aspatially filtered reference frame).

Color processing stage 312 may perform various operations associatedwith adjusting color information in the image data. The operationsperformed in color processing stage 312 include, but are not limited to,local tone mapping, gain/offset/clip, color correction,three-dimensional color lookup, gamma conversion, and color spaceconversion. Local tone mapping refers to spatially varying local tonecurves in order to provide more control when rendering an image. Forinstance, a two-dimensional grid of tone curves (which may be programmedby the central control module 320) may be bi-linearly interpolated suchthat smoothly varying tone curves are created across an image. In someembodiments, local tone mapping may also apply spatially varying andintensity varying color correction matrices, which may, for example, beused to make skies bluer while turning down blue in the shadows in animage. Digital gain/offset/clip may be provided for each color channelor component of image data. Color correction may apply a colorcorrection transform matrix to image data. 3D color lookup may utilize athree dimensional array of color channel output values (e.g., R, G, B)to perform advanced tone mapping, color space conversions, and othercolor transforms. Gamma conversion may be performed, for example, bymapping input image data values to output data values in order toperform gamma correction, tone mapping, or histogram matching. Colorspace conversion may be implemented to convert image data from one colorspace to another (e.g., RGB to YCbCr). Other processing techniques mayalso be performed as part of color processing stage 312 to perform otherspecial image effects, including black and white conversion, sepia toneconversion, negative conversion, or solarize conversion.

Output rescale module 314 may resample, transform and correct distortionon the fly as the ISP 206 processes image data. Output rescale module314 may compute a fractional input coordinate for each pixel and usesthis fractional coordinate to interpolate an output pixel via apolyphase resampling filter. A fractional input coordinate may beproduced from a variety of possible transforms of an output coordinate,such as resizing or cropping an image (e.g., via a simple horizontal andvertical scaling transform), rotating and shearing an image (e.g., vianon-separable matrix transforms), perspective warping (e.g., via anadditional depth transform) and per-pixel perspective divides applied inpiecewise in strips to account for changes in image sensor during imagedata capture (e.g., due to a rolling shutter), and geometric distortioncorrection (e.g., via computing a radial distance from the opticalcenter in order to index an interpolated radial gain table, and applyinga radial perturbance to a coordinate to account for a radial lensdistortion).

Output rescale module 314 may apply transforms to image data as it isprocessed at output rescale module 314. Output rescale module 314 mayinclude horizontal and vertical scaling components. The vertical portionof the design may implement series of image data line buffers to holdthe “support” needed by the vertical filter. As ISP 206 may be astreaming device, it may be that only the lines of image data in afinite-length sliding window of lines are available for the filter touse. Once a line has been discarded to make room for a new incomingline, the line may be unavailable. Output rescale module 314 maystatistically monitor computed input Y coordinates over previous linesand use it to compute an optimal set of lines to hold in the verticalsupport window. For each subsequent line, output rescale module mayautomatically generate a guess as to the center of the vertical supportwindow. In some embodiments, output rescale module 314 may implement atable of piecewise perspective transforms encoded as digital differenceanalyzer (DDA) steppers to perform a per-pixel perspectivetransformation between a input image data and output image data in orderto correct artifacts and motion caused by sensor motion during thecapture of the image frame. Output rescale may provide image data viaoutput interface 314 to various other components of system 100, asdiscussed above with regard to FIGS. 1 and 2.

In various embodiments, the functionally of components 302 through 342may be performed in a different order than the order implied by theorder of these functional units in the image processing pipelineillustrated in FIG. 3, or may be performed by different functionalcomponents than those illustrated in FIG. 3. Moreover, the variouscomponents as described in FIG. 3 may be embodied in variouscombinations of hardware, firmware or software.

Bayer Format to Four Color Plane Format

FIG. 4 is a block diagram illustrating conversion of image data fromBayer data format to four-plane data format, according to oneembodiment. The block diagram includes a planarizer circuit 410, amemory 420, a sensor interface circuit 430, and a color plane processingcircuit 440. The planarizer 410 receives image data in the Bayer dataformat and writes the image data in the four-plane data format. Theplanarizer 410 can also read the four-plane data and convert it to theBayer data format. The planarizer 410 may be located within memoryinterface module 326 described above with reference to FIG. 3 In someembodiments, the functionality of the planarizer circuit 410 may beimplemented in hardware, firmware, software, or some combinationthereof.

The planarizer 410 includes a data splitter/combiner circuit 412, a pairof DMA buffers 414 and 415, and a pair of DMA interfaces 416 and 417.The data splitter/combiner circuit 412 either splits the Bayer data intodifferent planes of data or alternatively combines the different planesof data into the Bayer data as described below.

In one embodiment where the planarizer 410 converts the image data fromthe Bayer data format to the four-plane data format, the planarizer 410receives the image data via the sensor interface 430. The sensorinterface 430 may be similar to the sensor interface 302 and may receivethe image data from either the image sensor 202 or from a memory (e.g.,system memory 230 or persistent storage 228). The received Bayer dataincludes a plurality of pixel rows and pixel columns, where the oddnumbered pixel rows include alternate Red and Green pixels, and the evennumbered pixel rows includes alternate Green and Blue pixels as shown inthe Bayer data 510 in FIG. 5. The planarizer 410 converts the Bayer datainto the four-plane data on a pixel row basis as described below.

For a given pixel row, the data splitter/combiner circuit 412 splits thepixel row data into pixel data of the same color. For odd pixel rows,Red pixel data is split from Green pixel data. For even pixel rows,Green pixel data is split from and Blue pixel data. The split pixel datais buffered in one of the DMA buffers 414 and 415.

DMA interface 416 is coupled to DMA channel 1 422 and reads and writesimage data to the memory 420 via DMA channel 1 422. DMA interface 417 iscoupled to DMA channel 2 424 and reads and writes image from to thememory 424 via DMA channel 2 424. DMA channel 1 422 and DMA channel 2424 represent separate physical channels through which data can betransferred with the memory 420. For example, DMA channel 1 may be oneset of wires and DMA channel 2 may be another set of wires.

When a complete row of image data has been stored into the DMA buffers414 and 415, the DMA interface 416 writes the image data in DMA buffer414 to memory 420. Also, DMA interface 417 writes the image data in DMAbuffer 415 to memory 420. In other embodiments, the DMA buffers 416 and417 may write data at other times, such as when the buffers 414 and/or415 store a threshold amount of data.

The process of splitting the pixel row data, buffering the split data,and writing the buffered data is repeated for every pixel row of theBayer data. In one embodiment, each of the buffers 414 and 415 has alarge enough to hold pixel data equal to one half of a pixel row ofimage data such that one full pixel row of image data may be written tomemory in each write operation.

Color plane processing circuit 440 reads the image data in thefour-plane data format from the memory 420 and performs furtherprocessing on the individual color planes. For example, the color planeprocessing circuit 440 applies one or more digital image processingalgorithms to one color plane of data at a time to process the imagedata further. The image processing algorithms are typically designed toprocess non-Bayer format data. Converting the Bayer format image intofour separate planes of data enables the color plane processing circuit440 to process image data one color plane at a time using thesealgorithms that are not compatible with Bayer data. After processing thefour-plane data, the color plane processing circuit 440 writes back theprocessed image data back to the memory 420 for additional imageprocessing. One of the advantages for converting Bayer data format tofour-plane data format is to make compression of image data easier.Compression formats, such as JPEG, generally do not support Bayer dataformat. By converting the Bayer data to four-plane data, processingimage data for data compression is easier as some GPU and CPU operationsare simplified while operating on planar data. As another example, aface detection algorithm might work well on a single plane of greenpixel data.

The planarizer 410 can also read the image data that is stored in thememory 420 in four-plane data format, and convert the data to the Bayerdata format. The planarizer 410 receives the image data from the memory420 over the DMA interfaces 416 and 417 via the DMA channels 422 and424. The data splitter/combiner circuit 412 accesses the datacorresponding to each plane of the four-plane data that is stored atvarious memory blocks to combine the data into the Bayer data format.

The process of converting the image data from the Bayer data format tothe four-plane data format and vice versa is further described belowwith reference to FIG. 5.

FIG. 5 is a block diagram illustrating additional details for convertingimage data from Bayer data format to four-plane data format, accordingto one embodiment. FIG. 5 shows the Bayer data 510, the four-plane data550, and the two DMA channels, DMA1 and DMA2, over which the data istransferred. FIG. 5 also shows two blocks within the memory 420, memoryblock 580 and memory block 590, where the four-plane data 550 is stored.The two memory blocks 580 and 590 are two separate blocks of memorylocated at two separate address spaces.

The Bayer data 510 includes repeating pixel groups, where each pixelgroup includes a 2×2 block pixels such as a Red pixel R 511, two Greenpixels Gr 512 and Gb 521, and a Blue pixel B 522. The Bayer data 510shown in FIG. 5 includes a plurality of pixel rows and pixel columns,where the odd numbered pixel rows include alternate Red and Greenpixels, and the even numbered pixel rows includes alternate Green andBlue pixels.

In one embodiment, the planarizer 410 converts the Bayer data 510 intofour-plane data 550 via two DMA channels, DMA1 and DMA2. For each pixelrow of the Bayer data 510, the planarizer 410 writes pixel datacorresponding to either Red and Green pixels or Green and Blue pixels totwo separate locations in a memory (e.g., memory 420) via DMA channels,DMA1 and DMA2. For odd numbered pixel rows, the planarizer 410 writespixel values corresponding to the Red pixels to a first memory block viaDMA1 channel. For example, Red pixels 511 and 513 of the first pixel roware written to consecutive memory locations of the first memory block580, and specifically to a first sub-block 581 within the first memoryblock 580. The planarizer 410 writes all Red pixel values of the firstpixel row to consecutive memory locations of the first memory sub-block581. For the odd numbered pixel rows, the planarizer 410 also writespixel values corresponding to the Green pixels to the second memoryblock 590 via DMA2 channel. For example, Green pixels 512 and 514 of thefirst pixel row are written to consecutive memory locations of thesecond memory block 590, and specifically to a first sub-block 591within the second memory block 590. The planarizer 410 writes all Greenpixel values of the first pixel row to consecutive memory locations ofthe first memory sub-block 591. The planarizer 410 writes the Red pixelsand the Green pixels of each odd numbered pixel rows simultaneously. Forexample, while the Red pixels are written via DMA1 channel to thesub-block 581, the Green pixels are simultaneously written via DMA2channel to the sub-block 591.

For even numbered pixel rows, the planarizer 410 writes pixel valuescorresponding to the Green pixels to the first memory block 580 via DMA1channel, and writes pixels corresponding to the Blue pixels to thesecond memory block 590 via DMA2 channel. For example, Green pixels 521and 523 of the second pixel row are written to consecutive memorylocations of a second sub-block 582 of the first memory block 580. Bluepixels 522 and 524 of the second pixel row are written to consecutivememory locations of a second sub-block 592 of the second memory block590. The planarizer 410 writes all Green pixel values of the secondpixel row to consecutive memory locations of the second memory sub-block582 and all Blue pixel values of the second pixel row to consecutivememory locations of the second memory sub-block 592. The planarizer 410writes the Green pixels and the Blue pixels of each even numbered pixelrows simultaneously, similar to the writing of odd numbered pixelsdescribed above. For example, while the Green pixels are written viaDMA1 channel to sub-block 582, the Blue pixels are simultaneouslywritten via DMA2 channel to sub-block 592.

The planarizer 410 repeats the process of converting the Bayer data 510into the four-plane data 550 by writing the data one pixel row at a timeuntil all pixel rows of the Bayer data 510 are converted into thefour-plane data 550. For example, the planarizer 410 writes data viaDMA1 channel to the sub-blocks 581, 582, 583, and 584 for the first fourpixel rows of the Bayer data 510, where the sub-blocks 581 through 584are all part of the first memory block 580. The planarizer 410 alsowrites data via DMA2 channel to the sub-blocks 591, 592, 593, and 594for the first four pixel rows of the Bayer data 510, where thesub-blocks 591 through 594 are all part of the second memory block 590.In practice, the number of sub-blocks included within each of the firstmemory block 580 and the second memory block 590 equals the number ofpixel rows of the Bayer data 510.

Each plane of image data spans multiple sub-blocks. Sub-blocks 581 and583 represent the red R plane of image data. Sub-blocks 582 and 584represent the green Gb plane of image data. Sub-blocks 591 and 593represent the green Gr plane of image data. Sub-blocks 592 and 594represent the blue B plane of image data.

Additionally, the planarizer 410 can read the four-plane data 550 fromthe memory via the two DMA channels, DMA1 and DMA2. The process ofreading the four-plane data 550 and converting the data into the Bayerdata 510 is inverse of the process of converting the Bayer data 510 tothe four-plane data 550 as described above. For example, the planarizer410 reads the image data at the first sub-block 581 of the first memoryblock 580 via DMA1 and reads the image data at the first sub-block 591of the memory block 590 via DMA2 to create the first pixel row of theBayer data 510.

The planarizer 410 reads the image data corresponding to all Red pixelsstored in the sub-block 581 via DMA1 in a single read operation. Theplanarizer 410 also reads the image data corresponding to all Green (Gr)pixels stored in the sub-block 591 via DMA2 in a single read operation.The image data of the sub-blocks 581 and 591 is buffered at buffers, DMAbuffer 414 and DMA buffer 416, via DMA interfaces 416 and 417 while theplanarizer 410 is reading the data. The buffered image data is used toreconstruct the first pixel row of the Bayer data 510 by usingalternating data from the two buffers. For example, odd position pixelsof the first pixel row of the Bayer data 510 are reconstructed from thepixel data of the buffered data at DMA buffer 414, which corresponds tothe Red pixels read from the sub-block 581. Even position pixels of thefirst pixel row of the Bayer data 510 are reconstructed from the pixeldata of the buffered data at DMA buffer 415, which corresponds to theGreen (Gr) pixels read from the sub-block 591.

Next, the planarizer 410 reads the image data at the second sub-block582 of the first memory block 580 via DMA1 and the second sub-block 592of the memory block 590 via DMA2 to create the second pixel row of theBayer data 510, similar to the first pixel discussed above. Theplanarizer 410 reads the image data corresponding to all Green (Gb)pixels stored in the sub-block 582 via DMA1 in a single read operation.The planarizer 410 also reads the image data corresponding to all Bluepixels stored in the sub-block 592 via DMA2 in a single read operation.The image data of the sub-blocks 582 and 592 is buffered at buffers, DMAbuffer 414 and DMA buffer 416, via DMA interfaces 416 and 417 while theplanarizer 410 is reading the data. The buffered image data is used toreconstruct the second pixel row of the Bayer data 510 by usingalternating data from the two buffers. For example, odd positionedpixels of the second pixel row of the Bayer data 510 are reconstructedfrom the pixel data of the buffered data at DMA buffer 414, whichcorresponds to the Green (Gb) pixels read from the sub-block 582. Evenpositioned pixels of the first pixel row of the Bayer data 510 arereconstructed from the pixel data of the buffered data at DMA buffer415, which corresponds to the Blue pixels read from the sub-block 592.The planarizer 410 repeats the process of reading image data stored inthe sub-blocks to create the pixel rows of the Bayer data 510 for allsub-blocks of the four-plane data 550.

FIG. 6 is a flowchart illustrating a method of converting image datafrom Bayer data format to four-plane data format, according to oneembodiment. The steps of the method 600 are performed by the planarizer410. In some embodiments, each step of the method 600 may be performedby a separate circuit or module within the planarizer 410.

The planarizer 410 receives 605 image data in a data format thatincludes multiple pixel groups, where each pixel group includes a firstpixel type, a second pixel type, a third pixel type, and a fourth pixeltype. For example, the received image data is in Bayer data format,where each pixel group includes Red, first Green Gr, Blue, and secondGreen Gb pixels such that the Red pixels are the first type of pixels(e.g., Red pixels 511 and 513) of FIG. 5, Green pixels Gr of odd pixelrows are the second type of pixels (e.g., Green pixels 512 and 514),Green pixels Gb of even pixel rows are the third type of pixels (e.g.,Green pixels 521 and 523), and the Blue pixels are the fourth type ofpixels (e.g., Blue pixels 522 and 524).

The planarizer 410 writes 610 image data to a memory via a first memorychannel and a second memory channel, where the pixels of the first type(e.g., R) and the third type (e.g., Gb) are written via the first memorychannel, and the pixels of the second type (e.g., R) and the fourth type(e.g., Gr) are written via the second memory channel. For example, theplanarizer 410 writes pixel data corresponding to the Red pixels 511 and513, and the Green pixels 521 and 523 via the first memory channel,DMA1. The planarizer 410 also writes pixel data corresponding to theGreen pixels 512 and 514, and the Blue pixels 522 and 524 via the secondmemory channel, DMA2. The planarizer 410 repeats the step 610 for allpixel rows of the Bayer data 510 as discussed above with reference toFIG. 5.

In some embodiments, the planarizer 410 also reads the four-plane data550 and converts 615 the data into the Bayer data 510. The planarizer410 reads the four-plane data 550 stored at memory 420 as four separateplanes of data. For example, the four-plane data 550 corresponding tothe first pixel row of the Bayer data 510 is stored at sub-blocks 581and 592 such that sub-block 581 stores Red pixel data and sub-block 591stores Green (Gr) pixel data. The four-plane data 550 corresponding tothe second pixel row of the Byer data 510 is stored at sub-blocks 582and 592 such that sub-block 582 stores Green (Gb) pixel data andsub-block 592 stores Blue pixel data. The planarizer 410 reads thestored four-plane data 550 in order to create pixel rows of the Bayerdata 510. To create the first pixel row of the Bayer data 510, theplanarizer 410 reads the image data corresponding to all Red pixelsstored in the sub-block 581 via DMA1 and reads image data correspondingto all Green (Gr) pixels stored in the sub-block 591 via DMA2. The readimage data from the sub-blocks 581 and 591 is buffered and used toreconstruct the first pixel row by using alternating data from the twobuffers as described above with reference to FIG. 5. The planarizer 410creates the second pixel row of the Bayer data 510 by reading the imagedata corresponding to all Green (Gb) pixels stored in the sub-block 582via DMA1 and image data corresponding to all Blue pixels stored in thesub-block 592 via DMA2. The read image data from the sub-blocks 582 and592 is buffered and used to reconstruct the second pixel row by usingalternating data from the two buffers as described above with referenceto FIG. 5. As described above with reference to FIG. 5, the planarizer410 repeats the process of reading image data stored in the sub-blocksto create the pixel rows of the Bayer data 510 for all sub-blocks of thefour-plane data 550.

FIGS. 7A-7C illustrate modified Bayer image data formats, according toone embodiment. The image data formats illustrated in FIGS. 7A-7C aremodified versions of the Bayer data format discussed above withreference to FIGS. 5 and 6. The image data in any of the modified Bayerdata formats depicted in FIGS. 7A-7C may be converted into thefour-plane data as described above with reference to FIGS. 5 and 6, forfurther image processing.

FIG. 7A depicts a modified Bayer format where the pixels of the thirdtype in the Bayer format (i.e., Green pixels of even pixel rows) arereplaced with a White pixel W. This image format is known as RGBW Bayer.Data corresponding to the White pixel is captured without a color filterand such data corresponds to light associated with all colors, asopposed to a Green pixel representing only Green color image data.Replacing the Green pixels of the even pixel rows with the White pixelsincreases the luminance of the images captured by the image sensor.Alternatively, the pixels of the second type (i.e., Green pixels of oddpixel rows) may be replaced with the White pixels.

FIG. 7B depicts a modified Bayer format where the pixels of the thirdtype in the Bayer format (i.e., Green pixels of even pixel rows) arereplaced with an infrared (IR) pixel. Data of the IR pixel representslight in the infrared spectrum. Alternatively, the pixels of the secondtype (i.e., Green pixels of odd pixel rows) may be replaced with the IRpixels.

FIG. 7C depicts a modified Bayer format where an order of the pixeltypes within the Bayer data format are modified. For example, while theoriginal Bayer data includes RGrRGrRGr pattern of pixels in odd pixelrows and GbBGbBGbB pattern of pixels in the even pixel rows, themodified Bayer format of FIG. 7C includes RGrRGrRGr pattern of pixels inthe odd pixel rows and GbBGbBGbB pattern of pixels in the even pixelrows. In alternate embodiments, other orderings of the pixel types maybe formed to modify the Bayer format further.

Three Sensor Data to Bayer Format

FIG. 8 is a block diagram illustrating conversion of image data fromthree sensor data format to Bayer data format, according to oneembodiment. The block diagram of FIG. 8 is similar to the block diagramof FIG. 4 except for the memory 820, sensors 810, and sensor interfacecircuit 830. The sensors 810 includes three individual color sensors: aRed sensor 812 for sensing Red image data, a Green sensor 814 forsensing Green image data, and a Blue sensor 816 for sensing Blue imagedata. Each color sensor may be implemented by a separate charge coupleddevice (CCD). A prism (not shown) receives incoming light and directsthe red, blue, and green colors toward the corresponding color sensors.The sensor interface circuit 830 is an interface that may besubstantially similar to the sensor interface circuit 430 describedabove with reference to FIG. 4. The sensor interface circuit 830receives image data from either the sensors 810 and sends the receivedimage data to the memory 820.

The image data format for images captured by the three sensors is shownin FIG. 9 as Red sensor data 910, Green sensor data 920, and Blue sensordata 930. The image data from each of the three sensors is stored in thememory 820 in separate memory blocks such as Red block 822 for Redsensor data, Green block 824 for Green sensor data, and Blue block 826for Blue sensor data via the sensor interface 830. The planarizer 410converts the image data in the three sensor data format to the Bayerdata format using two memory channels, DMA1 and DMA2, as described belowwith reference to FIG. 9. The resulting Bayer image can then be passedto another circuit stage that processes the image using image processingtasks that are specifically designed for use with Bayer images. As aresult, existing image processing techniques can be re-used to processimage data from a three color sensor camera.

FIG. 9 is a block diagram illustrating additional details for convertingimage data from three sensor data format to Bayer data format, accordingto one embodiment. FIG. 9 shows Red sensor data 910, Green sensor data920, and Blue sensor data 930, Bayer data 950, and the two DMA channels,DMA1 and DMA2.

The planarizer 410 converts the Red sensor data 910, the Green sensordata 920, and the Blue sensor data 930 to the Bayer data 950 via two DMAchannels, DMA1 and DMA2. The Bayer data 950 has the same data format asthat of the Bayer data 510 described above with reference to FIG. 5. TheBayer data 950 includes a plurality of pixel rows and pixel columns suchthat the odd numbered pixel rows include alternate Red and Green Grpixels, and the even numbered pixel rows includes alternate Green Gb andBlue pixels.

The planarizer 410 generates the first odd numbered pixel row (i.e.,first pixel row) of the Bayer data 950 as follows. The planarizer 410reads the pixel data corresponding to the first pixel row of the Redsensor data 910 and the first pixel row of the Green sensor data 920,and combines the data from the two pixel rows to generate the firstpixel row of the Bayer data 950 as described below. The DMA interface416 reads the pixel data corresponding to the first Red pixel 911 of theRed sensor data 910 via DMA channel DMA1. The data splitter/combinercircuit 412 then writes the pixel 911 as the first pixel of the firstpixel row of the Bayer data 950. The DMA interface 417 reads the pixeldata corresponding to the first Green pixel 921 of the Green sensor data920 via DMA channel DMA2. The data splitter/combiner circuit 412 thenwrites the pixel 921 as the second pixel of the first pixel row of theBayer data 950. The data splitter/combiner circuit 412 writes pixel datacorresponding to the first and second pixels of the first pixel row ofthe Bayer data 950 simultaneously via DMA channels, DMA1 and DMA2.

Next, the DMA interface 416 reads the pixel data corresponding to thesecond Red pixel 912 of the Red sensor data 910 via DMA channel DMA1.The data splitter/combiner circuit 412 then writes as the third pixel ofthe first pixel row of the Bayer data 950. The DMA interface 417 readsthe pixel data corresponding to the second Green pixel 922 of the Greensensor data 920 via DMA channel DMA2. The data splitter/combiner circuit412 then writes as the fourth pixel of the first pixel row of the Bayerdata 950. The data splitter/combiner circuit 412 writes pixel datacorresponding to the third and fourth pixels of the first pixel row ofthe Bayer data 950 simultaneously via DMA channels, DMA1 and DMA2. Theplanarizer 410 repeats the above-described steps of writing pixel datafor adjacent Red and Green pixels of the first pixel row of the Bayerdata 950 until the end of all pixel data corresponding to the firstpixel rows of the Red sensor data 910 and the Green sensor data 920.

The planarizer 410 generates the first even numbered pixel row (i.e.,second pixel row) of the Bayer data 950 as follows. The planarizer 410reads the pixel data corresponding to the first pixel row of the Greensensor data 920 and the first pixel row of the Blue sensor data 930, andcombines the data from the two pixel rows to generate the second pixelrow of the Bayer data 950 as described below. The DMA interface 416reads the pixel data corresponding to the first Green pixel 921 of theGreen sensor data 920 via DMA channel DMA1. The data splitter/combinercircuit 412 then writes as the first pixel of the second pixel row ofthe Bayer data 950. The DMA interface 417 reads the pixel datacorresponding to the first Blue pixel 931 of the Blue sensor data 930via DMA channel DMA2. The data splitter/combiner circuit 412 then writesas the second pixel of the second pixel row of the Bayer data 950. Thedata splitter/combiner circuit 412 writes pixel data corresponding tothe first and second pixels of the second pixel row of the Bayer data950 simultaneously via DMA channels, DMA1 and DMA2. Next, the DMAinterface 416 reads the pixel data corresponding to the second Greenpixel 922 of the Green sensor data 920 via DMA channel DMA1. The datasplitter/combiner circuit 412 then writes as the third pixel of thesecond pixel row of the Bayer data 950. The DMA interface 417 reads thepixel data corresponding to the second Blue pixel 932 of the Blue sensordata 930 via DMA channel DMA2. The data splitter/combiner circuit 412then writes as the fourth pixel of the second pixel row of the Bayerdata 950. The data splitter/combiner circuit 412 writes pixel datacorresponding to the third and fourth pixels of the second pixel row ofthe Bayer data 950 simultaneously via DMA channels, DMA1 and DMA2. Theplanarizer 410 repeats the above-described steps of writing pixel datafor adjacent Green and Blue pixels of the second pixel row of the Bayerdata 950 until the end of all pixel data corresponding to the firstpixel rows of the Green sensor data 920 and the Blue sensor data 930.

The planarizer 410 repeats the process of reading the pixel datacorresponding to each pixel row of the Red sensor data 910 and the Greensensor data 920, and combining such data to generate each odd pixel rowof the Bayer data 950 for all pixel rows of the Red sensor data 910 andthe Green sensor data 920. For example, the first pixel rows of the Redsensor data 910 and the Green sensor data 920 are combined to generatethe first pixel row of the Bayer data 950. The second pixel rows of theRed sensor data 910 and the Green sensor data 920 are combined togenerate the third pixel row of the Bayer data 950. This process isrepeated for all pixel rows of the Red sensor data 910 and the Greensensor data 920. Similarly, the planarizer 410 repeats the process ofreading the pixel data corresponding to each pixel row of the Greensensor data 920 and the Blue sensor data 930, and combining such data togenerate each even pixel row of the Bayer data 950 for all pixel rows ofthe Green sensor data 920 and the Blue sensor data 930. For example, thefirst pixel rows of the Green sensor data 920 and the Blue sensor data930 are combined to generate the second pixel row of the Bayer data 950.The second pixel rows of the Green sensor data 920 and the Blue sensordata 930 are combined to generate the fourth pixel row of the Bayer data950. This process is repeated for all pixel rows of the Green sensordata 920 and the Blue sensor data 930.

FIG. 10 is a flowchart illustrating a method of converting image datafrom three sensor data format to Bayer data format, according to oneembodiment. The steps of the method 1000 are performed by the planarizer410. In some embodiments, each step of the method 1000 may be performedby a separate circuit or module within the planarizer 410.

The planarizer 410 receives 1005 a first image that includes pixels of afirst type. For example, the first image can be an image captured by aRed sensor (e.g., Red sensor data 910) with the pixels of the first typebeing Red pixels.

The planarizer 410 receives 1010 a second image that includes pixels ofa second type. For example, the second image can be an image captured bya Green sensor (e.g., Green sensor data 920) with the pixels of thesecond type being Green pixels.

The planarizer 410 receives 1015 a third image that includes pixels of athird type. For example, the third image can be an image captured by aBlue sensor (e.g., Blue sensor data 930) with the pixels of the thirdtype being Blue pixels.

The planarizer 410 generates 1020 a Bayer format image from the firstimage, the second image, and the third image. The planarizer 410generates odd numbered pixel rows of the Bayer image by combining datacorresponding to pixel rows of the first image and the second image asdescribed above with reference to FIG. 9. The planarizer 410 generateseven numbered pixel rows of the Bayer image by combining datacorresponding to pixel rows of the second image and the third image asdescribed above with reference to FIG. 9.

In one embodiment, a representation of the image processor or componentswithin the image processor may be stored as data in a non-transitorycomputer-readable medium (e.g. hard disk drive, flash drive, opticaldrive). These representations may be, for example, behavioral level,register transfer level, logic component level, transistor level andlayout geometry-level descriptions of the image processor.

The disclosure herein has been described in particular detail withrespect to a few possible embodiments. Those of skill in the art willappreciate that other embodiments may be practiced. First, theparticular naming of the components and variables, capitalization ofterms, the attributes, data structures, or any other programming orstructural aspect is not mandatory or significant, and the mechanismsthat implement the invention or its features may have different names,formats, or protocols. Also, the particular division of functionalitybetween the various system components described herein is merelyexemplary, and not mandatory; functions performed by a single systemcomponent may instead be performed by multiple components, and functionsperformed by multiple components may instead performed by a singlecomponent.

Finally, it should be noted that the language used in the specificationhas been principally selected for readability and instructionalpurposes, and may not have been selected to delineate or circumscribethe inventive subject matter. Accordingly, the disclosure herein isintended to be illustrative, but not limiting, of the scope of theinvention, which is set forth in the following claims.

What is claimed is:
 1. An apparatus, comprising: an interface configuredto receive an image in a format that comprises repeating pixel groups,each pixel group spread across a plurality of corresponding pixel rowsand including: a first pixel type and a second pixel type of a firstpixel row; and a third pixel type and a fourth pixel type of a secondpixel row; a memory; a first memory channel connecting a circuit to thememory; a second memory channel connecting the circuit to the memory;and the circuit configured to write the image to the memory via thefirst memory channel and the second memory channel, pixels of the firstpixel type in the image written to the memory via the first memorychannel and pixels of the second pixel type in the image written to thememory via the second memory channel simultaneously at a first time,pixels of the third pixel type in the image written to the memory viathe first memory channel and pixels of the fourth pixel type in theimage written to the memory via the second memory channel simultaneouslyat a second time subsequent to the first time.
 2. The apparatus of claim1, wherein the format of the received image is a Bayer data format. 3.The apparatus of claim 1, wherein the first pixel type is a Red pixel,each of the second pixel type and the third pixel type is a Green pixel,and the fourth pixel type is a Blue pixel.
 4. The apparatus of claim 3,wherein odd numbered pixel rows of the plurality of corresponding pixelrows comprise alternating Red and Green pixels, and wherein evennumbered pixel rows of the plurality of corresponding pixel rowscomprise alternating Green and Blue pixels.
 5. The apparatus of claim 1,wherein the circuit is further configured to write odd position pixelsof each pixel row of the image to the memory via the first memorychannel and write even position pixels of each pixel row of the image tothe memory via the second memory channel.
 6. The apparatus of claim 1,wherein the memory further comprises: two separate address spaces, eachaddress space comprising a plurality of sub-blocks, each sub-block tostore a contiguous data corresponding to one pixel type of the imagewritten to the memory, the one pixel type comprising one of the firstpixel type, the second pixel type, the third pixel type, and the fourthpixel type.
 7. The apparatus of claim 1, wherein the circuit is furtherconfigured to: read pixels of the first pixel type and the third pixeltype via the first memory channel; read pixels of the second pixel typeand fourth pixel type via the second memory channel; and convert theread pixels to an image in the format that comprises repeating pixelgroups, wherein the format is a Bayer format.
 8. A method, comprising:receiving, by a circuit connected to a memory via a first memory channeland a second memory channel, image data corresponding to the image beingin a format that includes repeating pixel groups, each pixel groupspread across a plurality of corresponding pixel rows and including: afirst pixel type and a second pixel type of a first pixel row; and athird pixel type and a fourth pixel type of a second pixel row; andwriting, with the circuit, the image data to the memory via the firstmemory channel and the second memory channel, pixels of the first pixeltype in the image written to the memory via the first memory channel andpixels of the second pixel type in the image written to the memory viathe second memory channel simultaneously at a first time, pixels of thethird pixel type in the image written to the memory via the first memorychannel and pixels of the fourth pixel type in the image written to thememory via the second memory channel simultaneously at a second timesubsequent to the first time.
 9. The method of claim 8, wherein theformat of the received image is a Bayer data format.
 10. The method ofclaim 8, wherein the first pixel type is a Red pixel, each of the secondpixel type and the third pixel type is a Green pixel, and the fourthpixel type is a Blue pixel.
 11. The method of claim 10, wherein oddnumbered pixel rows of the plurality of corresponding pixel rowscomprise alternating Red and Green pixels, and wherein even numberedpixel rows of the plurality of corresponding pixel rows comprisealternating Green and Blue pixels.
 12. The method of claim 8, furthercomprising, by the circuit, writing odd position pixels of each pixelrow of the image to the memory via the first memory channel and writingeven position pixels of each pixel row of the image to the memory viathe second memory channel.
 13. The method of claim 8, wherein the memoryfurther comprises two separate address spaces, each address spacecomprising a plurality of sub-blocks, each sub-block to store acontiguous data corresponding to one pixel type of the image written tothe memory, the one pixel type comprising one of the first pixel type,the second pixel type, the third pixel type, and the fourth pixel type.14. The method of claim 8, further comprising, by the circuit: readingpixels of the first pixel type and the third pixel type via the firstmemory channel; reading pixels of the second pixel type and fourth pixeltype via the second memory channel; and converting the read pixels to animage in the format that comprises repeating pixel groups, wherein theformat is a Bayer format.